perf: enable brotli request compression by default - #1052
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1052 +/- ##
==========================================
+ Coverage 91.77% 91.92% +0.14%
==========================================
Files 50 51 +1
Lines 3222 3232 +10
==========================================
+ Hits 2957 2971 +14
+ Misses 265 261 -4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vdusek
marked this pull request as ready for review
July 16, 2026 11:34
Pijukatel
reviewed
Jul 17, 2026
Pijukatel
approved these changes
Jul 18, 2026
vdusek
added a commit
that referenced
this pull request
Aug 5, 2026
Pushing to a dataset serialized every item in its own
`asyncio.to_thread` hop (through Crawlee's `json_dumps`), so a push paid
one thread round-trip per item. That helper also pretty-prints with
`indent=2`, and each payload was UTF-8 encoded twice — once for the
per-item size check, once again while chunking.
`_check_and_serialize` and `_chunk_by_size` are replaced by a single
blocking `_serialize_chunk(items, offset)` that fills one size-bounded
JSON array and returns the next offset. `push_data` drives it with one
`asyncio.to_thread` per chunk and pushes each chunk before serializing
the next, so payloads are still streamed rather than all materialized up
front.
Serialization is now local and compact (`json.dumps(...,
separators=(',', ':'))`) rather than Crawlee's `json_dumps`, whose
`indent=2` is deliberate for the on-disk files people actually read.
Nothing reads the wire format. Dropping the indent also lets CPython
take the C encoder fast path, which it only does when `indent is None`.
Measured on 20k realistic items against a mocked API client:
| | time | payload |
| ------ | ------ | -------- |
| before | 1.081s | 10.78 MB |
| after | 0.075s | 9.34 MB |
The payload saving is shape-dependent — larger for small or deeply
nested items — and compounds with the brotli work in #1052.
Two spurious API calls disappear as a side effect: the old chunker
emitted an empty `[]` chunk whenever the first item of a chunk landed
within 2 bytes of the limit, and it pushed `[]` rather than nothing when
PPE charging drove the push limit down to 0.
Stored items are unaffected. The only observable difference is that raw
request bodies are no longer pretty-printed.
*✍️ Drafted by Claude Code*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables brotli compression as the default for all SDK/client ↔ platform communication.
apify-client v3 defaults to
gzip, so enabling brotli takes two parts:apify-client[brotli]extra (apify-client[brotli]>=3.1.0,<4.0.0) so the brotli backend is always installed. v3.1 is the first release to add request body compression.ApifyClientAsync(Actor.new_clientand the storage-client factory_create_api_client) now passcompression='brotli'. Since the SDK hard-depends onapify-client[brotli], the brotli backend is guaranteed present, so brotli is passed directly (no availability check).Tests: a wire-level test drives a real request through an SDK-created client against a local HTTP server and asserts the body goes out with
Content-Encoding: brand round-trips back throughbrotli.decompress. It's parametrized over both client-creation paths (_create_api_clientandActor.new_client).Closes #1045
✍️ Drafted by Claude Code